Additional findings (outside current diff — PR may have been updated during review):

• 🟡 benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh:190-203 — The B200 DEP recipe (tp:8, ep:8, dp-attn:true) computes MAX_NUM_SEQS=$((2*CONC)) unconditionally (line 206), but under DP-attention each of the $TP DP ranks is an independent vLLM engine that only sees roughly CONC/TP of the total concurrency, so this over-provisions --max-num-seqs and --max-cudagraph-capture-size by a factor of TP per rank. The sibling dsv4_fp4_b300_vllm.sh added in this same PR divides by TP for the DP-attention case (MAX_NUM_SEQS=$((2*CONC/TP))), so the B200 script should mirror that branch for consistency with the intended DEP recipe.

  Extended reasoning...

  The bug: In dsv4_fp4_b200_vllm.sh, MAX_NUM_SEQS=$((2 * CONC)) (line 206) is computed unconditionally with no branch on DP_ATTENTION. This value is then used both for --max-num-seqs and, in the same PR's new addition, for --max-cudagraph-capture-size (line ~233).

  Under DP-attention (DP_ATTENTION=true), PARALLEL_ARGS is set to --tensor-parallel-size 1 --data-parallel-size $TP (lines 187-189), meaning each of the $TP GPUs runs an independent vLLM engine (a DP rank), and vllm-router fronts them with --intra-node-data-parallel-size $TP and consistent_hash policy so that AgentX session trees are distributed across ranks. Each rank therefore sees only roughly CONC/TP sessions in steady state, not the full CONC. --max-num-seqs and --max-cudagraph-capture-size are per-engine settings, so the correct per-rank cap under DP-attention is ~2*CONC/TP, not 2*CONC.

  Why this is inconsistent, not hypothetical: this exact issue is handled correctly in the sibling script touched by the same PR, dsv4_fp4_b300_vllm.sh. It computes:

  if [ "$DP_ATTENTION" = "true" ]; then
      # The DEP source recipe enforces 2*CONC = DP_WORLD_SIZE*MAX_NUM_SEQS.
      MAX_NUM_SEQS=$((2 * CONC / TP))
  else
      MAX_NUM_SEQS=$((2 * CONC))
  fi
  

  along with a startup guard requiring 2*CONC to be evenly divisible by TP. The B200 script never got the equivalent branch, so it keeps using the pre-DEP aggregate formula for every topology, including the DEP8 tiers.

  Step-by-step proof: Take the B200 DEP8 tier in nvidia-master.yaml (tp: 8, ep: 8, dp-attn: true) at conc: 72, one of the values in the new conc-list.

  1. TP=8, CONC=72, DP_ATTENTION=true.
  2. PARALLEL_ARGS becomes --tensor-parallel-size 1 --data-parallel-size 8 — 8 independent DP-rank engines.
  3. vllm-router load-balances the 72 concurrent session trees across those 8 ranks via consistent hashing, so each rank's steady-state load is ~9 sessions (~18 with the intended 2x headroom).
  4. But MAX_NUM_SEQS=$((2*72))=144 is passed unconditionally to --max-num-seqs and --max-cudagraph-capture-size for every rank.
  5. Compare to the B300 sibling at the equivalent DEP4 tier: it would compute MAX_NUM_SEQS=2*72/4=36, i.e. correctly scaled to its DP world size.
  6. Net effect on B200: every rank is configured with an 8x oversized scheduler batch cap and cudagraph capture ceiling relative to its actual per-rank load.

  Impact: --max-num-seqs and --max-cudagraph-capture-size are upper bounds, not exact allocations — the scheduler and cudagraph bucketing only grow to whatever batch size actually shows up, and B200 (unlike B300, which now explicitly lists every integer capture size) only sets a single --max-cudagraph-capture-size ceiling, so vLLM's default bucket schedule still governs which sizes actually get captured. That means the practical effect is likely limited to a modestly wider capture-size ceiling and some extra scheduler headroom rather than 8x graph memory or a crash. It does not produce incorrect model outputs. However, it is a real fidelity gap relative to the DEP recipe this PR is trying to reproduce, and it makes the B200 DEP numbers not directly comparable to the correctly-scaled B300 DEP tier from the same PR.

  Fix: mirror the B300 branch — add an if [ "$DP_ATTENTION" = "true" ]; then MAX_NUM_SEQS=$((2 * CONC / TP)); else MAX_NUM_SEQS=$((2 * CONC)); fi, plus the same 2*CONC % TP == 0 startup guard B300 uses.

• 🔴 benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh:226-233 — The B200 agentic vLLM recipe (dsv4_fp4_b200_vllm.sh) dropped --tool-call-parser deepseek_v4 and --enable-auto-tool-choice from VLLM_CMD during the array rewrite, even though both flags are still present in the sibling dsv4_fp4_b300_vllm.sh (added in this same PR) and in every other agentic vLLM/SGLang recipe in this directory. Without these flags, vLLM will not enable auto tool-choice or parse DeepSeek-V4's tool-call output into structured tool_calls, which breaks (or silently invalidates) the AgentX tool-use trace-replay workload on B200 while B300 continues to work correctly.

  Extended reasoning...

  What broke: The diff for benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh rewrites the inline vllm serve ... \ invocation into a VLLM_CMD=(...) array. In that rewrite, two flags that existed pre-PR — --tool-call-parser deepseek_v4 and --enable-auto-tool-choice — are deleted (visible as - --tool-call-parser deepseek_v4 / - --enable-auto-tool-choice in the diff hunk) and never reappear anywhere else in the file. The post-PR VLLM_CMD for B200 keeps --tokenizer-mode deepseek_v4 and --reasoning-parser deepseek_v4, but the tool-call-specific flags are simply gone.

  Why this is asymmetric/accidental, not intentional: The sibling script dsv4_fp4_b300_vllm.sh, touched in this exact same PR, targets the same model (DeepSeek-V4-Pro), the same new nightly vLLM image, and the same FLASHINFER_MLA_SPARSE_DSV4 attention backend / deep_gemm_amxf4_mega_moe MoE backend — yet its rewritten VLLM_CMD retains both --tool-call-parser deepseek_v4 and --enable-auto-tool-choice (see the final VLLM_CMD block in dsv4_fp4_b300_vllm.sh). If dropping these flags were a deliberate consequence of the new image or attention backend, B300 would have dropped them too — it didn't. A repo-wide sweep of benchmarks/single_node/agentic/*.sh also shows every other agentic vLLM/SGLang recipe (dsv4_fp8_h200.sh, dsv4_fp4_mi355x_vllm.sh, kimik2.5_*, minimaxm3_*, etc.) passes --tool-call-parser, with tool-using ones also passing --enable-auto-tool-choice. B200 is the sole outlier.

  Code path that triggers it: These agentic recipes run AgentX's trace-replay workload via aiperf against /v1/chat/completions, replaying multi-turn tool-use sessions (tools + tool_choice). vLLM only performs server-side tool-call parsing and validates tool_choice="auto"/"required" requests when started with --enable-auto-tool-choice --tool-call-parser <parser>. There is no fallback for this in benchmark_lib.sh — tool-call handling is configured exclusively through these vllm serve CLI flags.

  Impact / step-by-step proof:

  1. dsv4_fp4_b200_vllm.sh launches vllm serve ... with the VLLM_CMD array shown in the diff — no --enable-auto-tool-choice, no --tool-call-parser.
  2. AgentX's trace replay sends chat-completions requests carrying a tools array and tool_choice: auto (that's the whole point of the agentic-coding benchmark).
  3. Without --enable-auto-tool-choice, vLLM either rejects those requests outright ("auto tool choice requires --enable-auto-tool-choice and --tool-call-parser") or, if it doesn't hard-reject, returns the model's tool invocation as raw unparsed text rather than a structured tool_calls field.
  4. Either outcome breaks the trace replay: outright request failures can trip aiperf's --failed-request-threshold and fail the sweep, or (if requests aren't rejected) the benchmark silently measures a non-functional/non-comparable agentic loop, since downstream turns in the trace depend on structured tool-call results.
  5. Meanwhile dsv4_fp4_b300_vllm.sh, edited in the same PR, keeps both flags, so B300 continues to correctly parse and act on tool calls — producing a divergence where one half of this PR's two new recipes is broken/invalid and the other is fine.

  Fix: Restore --tokenizer-mode deepseek_v4, --tool-call-parser deepseek_v4, --enable-auto-tool-choice, and --reasoning-parser deepseek_v4 together in the B200 VLLM_CMD, matching the B300 sibling's ordering/flags (dsv4_fp4_b300_vllm.sh lines ~225-227).